Performance of Cross Validation in Tree-Based Models

Cross Validation (CV) is widely used to measure the performance of a classifier. The main purpose of this study is to explore the behavior of CV in tree-based models. We report experimental studies that compare a cross-validated tree classifier with an oracle classifier that is ideally derived on the knowledge of underlying distributions. The main observation of this study indicates that the difference between the testing and training error from a cross-validated tree classifier and an oracle classifier empirically has a linear regression relation. The “slope” and the “R2” of regression models are employed as the performance measures of a cross-validated tree classifier. Moreover, simulation reveals that the performance of a cross-validated tree classifier depends on the geometry, the parameters of the underlying distributions, and sample size. Such observations can explain and justify the behavior of CV in tree-based models.